{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Together api with Mixtral\n",
    "\n",
    "### Checkout my [Twitter(@rohanpaul_ai)](https://twitter.com/rohanpaul_ai) for daily LLM bits"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# !pip install together python-dotenv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import together\n",
    "import dotenv\n",
    "import os\n",
    "\n",
    "dotenv.load_dotenv()\n",
    "together.api_key = os.getenv(\"together_key\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_list = together.Models.list()\n",
    "print(f\"{len(model_list)} models available\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "https://pypi.org/project/together/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Will print something like below\n",
    "\n",
    "```\n",
    "120 models available\n",
    "\n",
    "['EleutherAI/gpt-j-6b',\n",
    " 'EleutherAI/gpt-neox-20b',\n",
    " 'EleutherAI/pythia-12b-v0',\n",
    " 'EleutherAI/pythia-1b-v0',\n",
    " 'EleutherAI/pythia-2.8b-v0',\n",
    " 'EleutherAI/pythia-6.9b',\n",
    " 'HuggingFaceH4/starchat-alpha',\n",
    " 'NousResearch/Nous-Hermes-13b',\n",
    " 'NousResearch/Nous-Hermes-Llama2-13b',\n",
    " 'NumbersStation/nsql-6B']\n",
    "```\n",
    "\n",
    "The `Complete` class of the Together Python Library allows you to easily integrate the Together API's completion functionality into your applications, allowing you to generate text with a single line of code.\n",
    "\n",
    "https://docs.together.ai/docs/python-complete\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = \"mistralai/Mixtral-8x7B-v0.1\"\n",
    "\n",
    "prompt = \"\"\"To install PSU in your desktop machine first you will\"\"\"\n",
    "\n",
    "output = together.Complete.create(\n",
    "  prompt = prompt,\n",
    "  model = model,\n",
    "  max_tokens = 64,\n",
    "  temperature = 0.7,\n",
    "  top_k = 50,\n",
    "  top_p = 0.7,\n",
    "  repetition_penalty = 1,\n",
    "  #stop = [] # add any sequence you want to stop generating at.\n",
    ")\n",
    "\n",
    "# print generated text\n",
    "print(output['output']['choices'][0]['text'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`max_tokens (integer, optional)` -- Maximum number of tokens the model should generate. Default: 128\n",
    "\n",
    "`stop (List[str], optional)` -- List of stop words the model should stop generation at. Default: [\"<human>\"]\n",
    "\n",
    "\n",
    "`temperature(float, optional)` -- A decimal number that determines the degree of randomness in the response. Default: 0.7\n",
    "\n",
    "`repetition_penalty (float, optional)` -- A number that controls the diversity of generated text by reducing the likelihood of repeated sequences. Higher values decrease repetition. Default: 1\n",
    "\n",
    "-----------------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Run Mixtral-8x7B - with @togethercompute API 🚀\n",
    "\n",
    "Streaming tokens instead of waiting for the entire response\n",
    "\n",
    "\n",
    "Use the `stream_tokens` parameter to enable streaming responses.\n",
    "\n",
    "When `stream_tokens` is true, in the request payload, the API returns events as it generates the response instead of waiting for the entire response first."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "import requests\n",
    "import sseclient\n",
    "\n",
    "model_name = \"mistralai/Mixtral-8x7B-v0.1\"\n",
    "\n",
    "def stream_tokens_from_api(prompt, api_key, model=model_name, max_tokens=512):\n",
    "    url = \"https://api.together.xyz/inference\"\n",
    "    headers = {\n",
    "        \"accept\": \"application/json\",\n",
    "        \"content-type\": \"application/json\",\n",
    "        \"Authorization\": f\"Bearer {api_key}\",\n",
    "    }\n",
    "    payload = {\n",
    "        \"model\": model,\n",
    "        \"prompt\": prompt,\n",
    "        \"max_tokens\": max_tokens,\n",
    "        \"temperature\": 0.7,\n",
    "        \"top_k\": 50,\n",
    "        \"top_p\": 0.7,\n",
    "        \"repetition_penalty\": 2,\n",
    "        \"stream_tokens\": True,\n",
    "    }\n",
    "\n",
    "    try:\n",
    "        response = requests.post(url, json=payload, headers=headers, stream=True)\n",
    "        response.raise_for_status()\n",
    "    except requests.RequestException as e:\n",
    "        raise RuntimeError(f\"Request to API failed: {e}\")\n",
    "\n",
    "    try:\n",
    "        client = sseclient.SSEClient(response)\n",
    "        for event in client.events():\n",
    "            if event.data == \"[DONE]\":\n",
    "                break\n",
    "            yield json.loads(event.data)[\"choices\"][0][\"text\"]\n",
    "    except Exception as e:\n",
    "        raise RuntimeError(f\"Error while streaming tokens: {e}\")\n",
    "\n",
    "# Usage Example\n",
    "api_key = \"YOUR_API_KEY\"  # Replace with your API key\n",
    "prompt = \"To install PSU in your desktop machine first you will\"\n",
    "for token in stream_tokens_from_api(prompt, api_key):\n",
    "    print(token, end=\"\", flush=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "📌 Usage Example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "api_key = \"YOUR_API_KEY\"\n",
    "prompt = \"To install PSU in your desktop machine first you will\"\n",
    "\n",
    "for token in stream_tokens_from_api(prompt, api_key):\n",
    "    print(token, end=\"\", flush=True)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
